Generalization to novel object configurations and instances across diverse tasks and environments is a critical challenge in robotics. Keypoint-based representations have been proven effective as a succinct representation for capturing essential object features, and for establishing a reference frame in action prediction, enabling data-efficient learning of robot skills. However, their manual design nature and reliance on additional human labels limit their scalability. In this paper, we propose KALM, a framework that leverages large pre-trained vision-language models (LMs) to automatically generate task-relevant and cross-instance consistent keypoints. KALM distills robust and consistent keypoints across views and objects by generating proposals using LMs and verifies them against a small set of robot demonstration data. Based on the generated keypoints, we can train keypoint-conditioned policy models that predict actions in keypoint-centric frames, enabling robots to generalize effectively across varying object poses, camera views, and object instances with similar functional shapes. Our method demonstrates strong performance in the real world, adapting to different tasks and environments from only a handful of demonstrations while requiring no additional labels. Website: https://kalm-il.github.io/

This paper presents an innovative application of Transformer-XL for long sequence tasks in robotic learning from demonstrations (LfD). The proposed framework effectively integrates multi-modal sensor inputs, including RGB-D images, LiDAR, and tactile sensors, to construct a comprehensive feature vector. By leveraging the advanced capabilities of Transformer-XL, particularly its attention mechanism and position encoding, our approach can handle the inherent complexities and long-term dependencies of multi-modal sensory data. The results of an extensive empirical evaluation demonstrate significant improvements in task success rates, accuracy, and computational efficiency compared to conventional methods such as Long Short-Term Memory (LSTM) networks and Convolutional Neural Networks (CNNs). The findings indicate that the Transformer-XL-based framework not only enhances the robot's perception and decision-making abilities but also provides a robust foundation for future advancements in robotic learning from demonstrations.

The impressive generalization capabilities of large neural network models hinge on the ability to integrate enormous quantities of training data. This presents a major challenge for most downstream tasks where data is scarce. As a result, we have seen a transformation over the years away from training large models entirely from scratch, and toward methods that utilize finetuning or few-shot learning. Classically, models might be pre-trained on a large-scale supervised or self-supervised task (e.g., pre-training a large ResNet model on ImageNet), and then the last few layers of the model might be fine-tuned on a much smaller dataset for the task of interest. More recently, open-vocabulary vision-language models and promptable language models have made it possible to avoid fine-tuning, and instead define new tasks by constructing a textual prompt, potentially containing a few examples of input-output pairs.

The new technology comes from computer scientists based at University of California - Berkeley. Taking'machine learning' principles and creating specialized'robotic learning' systems, the researchers have given robots a degree of precognition. This new way of thinking will, one day, help to advance self-driving cars and to develop more intelligent robotic assistants for business operations. As things stand currently, the new technology has been tested out through an initial prototype which focuses on learning simple manual skills entirely from autonomous play. This is the foundation for more advanced applications with robotics.

The robots of the world are uniting – and that's either a great thing or a terrifying thing depending on your view. Google has a plan to speed up robotic learning, and it involves getting robots to share their experiences – via the cloud – and collectively improve their capabilities – via deep learning. Sergey Levine from the Google Brain team, along with collaborators from Alphabet subsidiaries DeepMind and GoogleX, published a blog post on Monday describing an approach for "general-purpose skill learning across multiple robots." Teaching robots how to do even the most basic tasks in real world settings such as homes and offices has vexed roboticists for decades. To tackle this challenge, the Google researchers decided to combine two recent technology advances.

Google has a plan to speed up robotic learning, and it involves getting robots to share their experiences and collectively improve their capabilities. Sergey Levine from the Google Brain team, along with collaborators from Alphabet subsidiaries DeepMind and X, published a blog post on Monday describing an approach for "general-purpose skill learning across multiple robots." Teaching robots how to do even the most basic tasks in real-world settings like homes and offices has vexed roboticists for decades. To tackle this challenge, the Google researchers decided to combine two recent technology advances. The first is cloud robotics, a concept that envisions robots sharing data and skills with each other through an online repository.